TRANSCODERS AND METHODS 

CROSS-REFERENCE TO RELATED APPLICATIONS 

The following cofiled US patent applications disclose related subject matter: 
Appl.Nos.: 10/..., filed 10/..., filed ... The following copending US patent 
application discloses related subject matter: Appl. No.: 09/089,290, filed 6/1/1998. All 
of these referenced applications have a common assignee with the present 
application. 

BACKGROUND OF THE INVENTION 

The present invention relates to digital video image processing, and more 
particularly, to methods and systems for transcoding from one video format to 
another with differing resolution. 

Currently, a large body of video content exists as MPEG-2 encoded bitstreams 
ready for DVD or broadcast distribution. This MPEG-2 content is usually available at 
a high bitrate (e.g., 6 Mbps), in interlaced SDTV (standard definition television) format 
(704x480 pixels). However, for effective video transmission, many applications such 
as 3G wireless infrastructure, video streaming, home networking, et cetera use low 
bitrate, progressive standards such as MPEG-4 or H.263. Due to the potential high- 
volume market associated with these applications, video transcoding which can 
convert MPEG-2 bitstreams into MPEG-4 bitstreams is an important, emerging 
technology. 

Figure 2a shows generic DCT-based motion-compensated encoding which is 
used in MPEG-2 and MPEG-4. Figure 2b illustrates a straightforward, but 
computationally intensive, resolution-reducing transcoder for conversion of an MPEG- 
2 bitstream into a lower-resolution MPEG-4 bitstream; the first row of operations 
decodes the input MPEG-2 bitstream, the middle operation down-samples the 
reconstructed video frames by a factor of two in both vertical and horizontal 
dimensions, and the bottom row performs MPEG-4 encoding. In particular, the input 
MPEG-2 SDTV bitstream is decoded by a conventional decoder that performs 
Variable-Length Decoding (VLD), Inverse Quantization (IQ), Inverse Discrete Cosine 
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Transform (IDCT), and Motion Compensation (MC) to produce SDTV-resolution raw 
frames in the 4:2:0 format. Spatial down-sampling by a factor of two is then 
performed vertically and horizontally to produce raw frames. Spatial downsampling 
along the vertical dimension is performed by extracting the top field of the raw 
interlaced SDTV frame. Spatial downsampling along the horizontal dimension is 
subsequently implemented either by discarding odd-indexed pixels or by filtering 
horizontally with the [1 ; 1] kernel and then discarding the odd-indexed pixels. Th.s 
spatial downsampling yields raw frames at the resolution 352x240. These frames are 
converted to CIF resolution by appending a 352x48 block of zeros to each raw frame. 
Next the CIF-resolution raw frames are input to an MPEG-4 encoder that performs 
Motion Estimation (ME), Discrete Cosine Transform (DCT), Quantization (Q) and 
Variable-Length Coding (VLC) to obtain the transcoded MPEG-4 CIF bitstream. 

However, because the CIF-resolution frames are obtained from down- 
sampling the SDTV-resolution frames, the motion field described by the MPEG-4 
motion vectors is a downsampled version of the motion field described by the MPEG- 
2 motion vectors. This implies that the ME stage may be eliminated in Figure 2b 
because MPEG-2 motion vectors may be re-used in the MPEG-4 encoder, as 
suggested in Figure 3a. In fact, if the ME utilizes an exhaustive search to determine 
the motion vectors, then it consumes approximately 70% of the MPEG-4 encoder 
cycles. In this case, elimination of the ME stage by estimating the MPEG-4 motton 
vectors from the MPEG-2 motion vectors will significantly improve transcoding 
performance. 

Now, every MPEG-2 frame is divided into 1 6x1 6 MacroBlocks (MBs) with the 
16x16 luminance pixels subdivided into four 8x8 blocks and the chrominance pixels, 
depending upon format, subsampled as one, two, or four 8x8 blocks; the DCT is 
performed on 8x8 blocks. Each macroblock is either intra- or inter-coded. The 
spatial downsampler of Figure 3a converts a "quartet" of four MBs that are co-located 
as shown in Figure 3b into a single 16x16 Macroblock that will be MPEG-4 encoded. 
Each inter-coded MB is associated with a motion vector that locates the reference 
macroblock in a preceding anchor-frame. Therefore, every MB quartet has four 
associated MPEG-2 motion vectors as shown in Figure 3c. And the prediction errors 
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from use of the reference macroblock as the predictor is DCT transformed; for 
luminance either as four 8x8 blocks according to spatial location (frame-DCT) or as 
four 8x8 blocks with two 8x8 blocks corresponding to the top field of the MB and two 
8x8 blocks corresponding to the bottom field of the MB (field-DCT). 

To eliminate the MPEG-4 ME stage in the Figure 2b baseline transcoder, 
estimate the MPEG-4 motion vector from the four associated MPEG-2 motion 
vectors as shown in Figure 3c. (Note that in B-frames, an MB may also have an 
additional motion vector to locate a reference macroblock in a subsequent anchor- 
frame.) And various motion vector estimation approaches have been proposed; for 
example Wee et al., Field-to-frame transcoding with spatial and temporal 
downsampling, IEEE Proc. Int. Conf. Image Processing 271 (1999) estimate the 
MPEG-4 motion-vector by testing each of the four scaled MPEG-2 motion vectors 
associated with a macroblock quartet on the decoded, downsampled frame that » 
being encoded by the MPEG-4 encoder. The tested motion vector that produces the 
least residual energy is selected as the estimated MPEG-4 motion vector. 

For the transcoder in Figure 3a, the input and output bitstreams are both 
coded quantized DCT coefficients. However, after the IDCT stage, spatial-doma.n 
processing accounts for most of the intermediate processing. Finally, the DCT stage 
returns the spatial-domain pixels to the frequency-domain for quantization and VLC 
processing. Some researchers suggested that the intermediate processing can be 
performed in the frequency domain, thus eliminating the IDCT and DCT stages in the 
transcoder. For example, Assuncao et al, A Frequency-Domain Video Transcoder for 
Dynamic Bit-Rate Reduction of MPEG-2 Bit Streams, 8 IEEE Trans. Cir. Sys. V.deo 
Tech. 953 (1998). 

And Merhav et al, Fast Algorithms for DCT-Domain Image Down-Sampl.ng 
and for Inverse Motion Compensation, 7 IEEE Tran. Cir. Sys. Video Tech. 468 
(1997), provides matrices for downsampling and inverse motion compensate in the 
frequency domain together factoring of the matrices for fast computations. 

Further, Song et al, A Fast Algorithm for DCT-Domain Inverse Mot.on 
Compensation Based on Shared Information in a Macroblock, 10 IEEE Trans. C.r. 
Sys Video Tech 767 (2000), disclose inverse motion compensation tak.ng advantage 
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of the adjacent ,oca.ions of .he four reference 8x8 blocks of a predicted macroblock 

to simplify the computations. 

Subsequently, Liu e. al. Local Bandwidth Constrained Fast Inverse Mo ton 
Condensation for DCT-Domain Video Transcoding, 12 IEEE TranXir Sys V,deo 
Tech 309 (2002) and A Fast and Memory Efficient Video Transcoder for Low Btf 
Rate Wireless Communications, IEEE Proc. Int. Conf. ASSP 1969 (2002), 
demonstrated reduced-complexity frequency-domain transcoding by downsampltng 
prior to inverse moton compensation in the frequency domam. 

Aral et al A Fast DCT-SQ Scheme for Images, 71 Trans. IEICE 1095 (1988), 
provides a factorization for the 8x8 OCT matrix which allows for fast computers. 

Hou A Fast Recursive Algorithm for Computing the Discrete Costne 
Transform, 35 IEEE Tran. ASSP 1455 (1987), provides a recursive method for the 
OCT analogous to the fas. Fourier transform (FFT, in which a 2N-poin. transform is 
expressed in terms of N-point transforms together with simple operates. 

SUMMARY OF THE INVENTION 

The present inventions provide resolution-reducing transcodtng methods 
including motion vector reuse by best predictor selecfion, motion vector 
search window adaptation to reference block bounded alignment, frequency domatn 
downsampling with frame-DCT blocks spaUally averaged but fleld-DCT blocks 
spatially averaged only horizontally and the field averaged, and mixtures of one- 
dimensional de-interlacing IDCT with IDCT plus downsampling. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The drawings are heuristic for clarity. 
Figures 1a-1d are flow diagrams. 

Figures 2a-2b show motion compensation encoding and a transcoder. 

Figures 3a-3d illustrate a transcoder and motion vector estimabon. 

Figures 4a-4b show transcoders. 

Figures 5a-5c illustrates motion vector refinement. 

Figure 6 is another transcoder. 
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DESCRIPTION OF THE PREFERRED EMBODIMENTS 



The preferred embodiment methods and systems oonvert MPEG-2 bitstreams 
into MPEG-4 bitstreams with spaflal-resolution reduction by downsampUng. The 
Lhods inolude re-use of mobon vectors for downsampled blocKs by scalrng th best 
predictor of four motion vectors prior to downsampling, refinement of motion vector 
estimates in the frequency domain by search windows which adapt to targe, and 
reference blocK boundary alignment, B-picture and l-/P-pic*ure separate 
downsampling methods, and mixture of de-interlacing one-d,mens,onal 1-D) rnverse 
DOT (IDCT) and 1-D IDCT plus downsampling together with inverse mobon 
compensabon after horizontal downsampling but prior to vertical downsampling ,n 
order to minimize drift. 

2 Motion Vector Estimation 

To describe the preferred embodiment motion vector esf.mat.on for 
transcoding MPEG-2 to MPEG-4, first briefly consider foliowing five prior art 
approaches: 

,1) random motion-vector estimation: The simplest motion-vector esbmabon 
algorithm for downsampled frames is the random algorithm proposed by Shanableh 

^Heterogeneous Video Transcoding to Lower Spatlo-Tempora, Resolutions and 
Different Enling Formats, 2 IEEE Trans. On Multimedia 1927 (2000). To obta, 
the MPEG-4 estimate, the algorithm randomly selects one of the four MPEG- motion 
vectors in Figure 3c and then halves its horizontal and vertical components. This 

difference between the MPEG-2 frame and the MPEG-4 frame. If a processo do* t . 
used to select a random number, then the random mobon-vector esbmabon algonthm 
has a very low cycle count. 

(2) average motion-vector estimation: Shen et at. Adaptive Mohon-Vecto 
Resampling for Compressed Video Downscaling, 9 IEEE Trans. Cir. Sys. V,deo 
Tech 929 (1999, and Shanableh et al, supra, proposed that the MPEG-4 mobon- 
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vector estimate may be obtained by separate averaging of *e honzonta, and vertca 
Ipc nts ot me tour MPEG-2 motion vectors. The averaged mo«on vector ,s then 
ri d to account for me spatial-resoiution redu*n. Tbis aigorifhm consumes 6 



adds and 2 shifts. Vi . , 

(3) w eigh,ed-average mo«on-vector es«ma,ion: Sben e, a,., sup a. an Ym - * 
V Jo Transcoding by Reducing Spatia, Resolution, IEEE Proc. Int. CortU image 
Processing 972 (2000) showed .ha. the performance of the average mot,on vector 
esCTn algolrn may be improved by adaptiveiy weighing the average so as to 
move me estimate toward motion vectors associated with MBs °° '^J^ 
The cycle count for this algorithm is 76 adds and two shifts, assumrng that 25 * of 
DCT terms in me four MPEG-2 macroblocks are non-zero. atedthatthe 

,4) median motion-vector estimation: Shanableh e, al, supra, demons, ated that 
median o, me four MPEG-2 motion vectors may be used as the MPEG-4 motion 
"e r e mate. The median is obtained by first calculating the distance between 
IITmTeG-2 motion vector and the res,. Next, the median motion vector ,s de^e 
e vector that has the leas, distance from the others. Finally, me medrar , m* on 
!r is scaled to obtain the MPEG-4mo S on-vector estimate^ ^^ZT 
vector estimation algorithm requires 30 adds, f 2 multiplies, two shrfts and three 



7— -norm motion-vector es,ima«on: Wee a, a, cited in the , bac^und. 
estimate me MPEG-4 motion-vector by testing each of the four scaled MPEG-2 
Ion Irs associated with a macroblock guarte, on the decoded, down-samp e d 
le which is being encoded by me MPEG-4 encoder. The teste mofon ve.. 
« produces me leas, residua, energy is selected as ,he eshmated 
vector. The cycle coun, for mis algorithm is 256 adds, three compares and two 

Shif,S ' The firs, preferred embodiment mo«on vector estimation mefttod is a fas, 
minimum-norm motion-vector es,ima«on which may be used in .ranscoder. , M 

Fiaures 3b-3c for a quartet of MPEG-2 macroblocks there are four MPEG-2 mot,on 

one ^jl* * ~— * - «« — k - For each of 
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mo ,ion vector points to a reference blocx that is an effective P»**££ 
ma crob.oc, Ne,, select among these four MPEG-2 motion v^Je on 
assoctated with the macroblocx having the smallest f norm. Then halv ft s mo 
vector to account for the resampling resolution reducbo, and 

♦ho Parnate for the MPEG-4 motion vector; see Figure 1a. Note 

the OCT residual blocKs due to quantization; and these enfnes are made 
entries decoder's VLD operation. This preferred embodiment 



non-zero 



Lore mathemafica, terms me foregoing can be deschbed as follow. Firs, 

were MPEG-2 compressed to yield the four motion vectors v„ n v, v., together w, 
Were _ ,,„ ri R*R DCTs- the number of DCTs depends upon the 

the corresponding quantized 8x8 DCTs, tne num 

which minimizes the 16x16 prediction error * - Now, 

corresponding quantized 8x8 DCTs, E„* ^ «* are 

were generated by the MPEG-2 motion compensation and compression. 

Next, downsample the quartet of (reconstructed, macroblocKs „. * * 
a fectorof 2 in each dimension to yield a singie macroblocK * which ,s to be MPEG-4 
massed. Preferably, the downsampiing occurs in the frequency domain. The 
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MPEG-4 compression includes finding a motion vector, v, for , which locates a 16x16 
luminance prediction f from a prior reconstructed reference frame. 

The preferred embodiment method estimates this motion vector v by the 

following steps. ^ IIB? .,2 _ 

(i) Compute the four squared norms l|£,|l 2 . Il&ll. Ilftll • I W where - 
ll£n ill 2 ♦ IM* ^ VMf + VMf with IM 2 = W W *• «" of squares of 
21 eilenl of ,„., Due to quantization, a large number of the 64 elements 

1) Pick n so that mf * me smallest of the four squared norms from step (i)_ 

ii) Estimate the motion vector v by v„/2 where n was determined ,n step (.,). Thus 
when v„ has half-pixel accuracy, v will have quarter-pixel accuracy. Of course, 
lall-pixel motion vectors corresponds to a prediction block resulting from „near 
interpolation of the closest integer-pixel motion vector located blocks. 

Note that ,he and the v„ are available mom me input MPEG-2 compress,on 
of the quartet of macroblocks, so the computations have low complex,*. 

Of course, the chrominance parts of a maoroblock use the motion vector 
derived from the luminance part, so there is no further motion vector to 
Also field rather than frame compression may generate two motion vectors, but M 
1 field motion vector as in the foregoing. And if one (or more, of the qua rto, o 
m acrob,ocks is skipped or no, encoded, then its correspond^ E„ w,ll be a Os and 
nave the smallest squared norm in step (ii); Ihus the computation of step (0 can be 
skipped. Lastiy, B-piCures have been omitted to reduce bttrate, but the same 
preferred embodiment methods could apply to the motion vectors for B-prCures. 

Variations of the preferred embodiment motion vector estimation methods 
include use of a different magnitude measure in place of the squared norm to 
m easure the magnitude of me DOT of the prediction errors, such as f norms 
although me OCT is not an isometry with respect to such norms for p * 2. Further, 
NxN arrays of macoblocks for downsampling by a factor of N in each dimension 
could be used with N greater than 2; and then the minimum-norm motion vector 
components are divided by N. Figure 1a illustrates the methods. 
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3 Motion Vector Estimation Experimental Results 

To compare the performance of the preferred embodiment mot.cn veco 
estimation with the various other motion-vector estimation methods, each of the 
^««^ln1h.t«n«*ofFN^3.. Then the PSNR,oss/frame 

output in the baseline transcoder of Figure 2b was evaluated. The average PSNR 
loss per frame (in dB) for the methods were as follows. 

Random 5.62 

Average 

Weighted average 7.4b 
Median "J- 34 
Minimum norm 0 
Preferred embodiment 0.58 

The median, minimum-norm and preferred embodimen, 

performance. Based on ,he cycle counts provided for .he methods, order .hese three 
algorithms from lowest to highest computation^ complexity as follows, median 
preferred embodiment < minimum-norm. Because the minimum norm method has 

ry high compute, complexity, the median and the preferred embod.men. 
Zionlctor estimation methods provide the best performance w,«h a Irade-oft o, 
low complexity (median) for accuracy (preferred embodiment). 

4 Frequency-Domain Transcoding 

Figure 3a shows the transcoder input and output bitstreams are coded, 
guantized OCT coefficients. However, after the IDCT stage, stage 
processing accounts for most o. the intermediate processing. Finafty, .he OCT stage 
returns .he spatial-domain pixels .0 the frequency domain (DOT domain) or 
quan.fca.ion and VLC processing. Prior researchers such as Chang ^ 
Manipulation and Compositing of MCDCT Compressed Vrdeo, 13 IEEE J. Sel. Areas 
Col (1995), Assuncao e. al, Transcoding o, MPEG-2 Video in ftre Frequency- 
Ool. IEEE Proc. inl. Com. ASSP 2633 (1997,, and Merhav e. aU = he 
background, suggested that .he intermediate processing can be performed m the 
bacKgrouno, ga transcoder, and 

frequency domain, thus eliminating the IDCT ana uo sua 



TI-35495/35498/355 14/3551 5/35536/35537 P.ge 9 



me preferred embodiments extend such methods. Thus first consider these prior 
frequency-domain transcoding methods. 

Chang et at, Manipulation and Compositing of MC-DCT Compressed Video, 
13 IEEE J Sel Areas Comm. 1 (1995), showed that motion compensation can be 
performed in the frequency domain (DCT-domain). Their algorithm was improved 
upon by Merhav et al and Assuncao et al, both cited in the background, who showed 
in addition that frequency domain motion compensation may be used in a frequency- 
domain transcoder. However, unlike the baseline transcoder in Figure 2b, the 
transcoder of Assuncao et al provided nitrate reduction but did not perform a spattal- 
resolu«on reduction. Subsequently, Lin e. al, Fast Algorithms for DCT-Domain Video 
Transcoding, IEEE Proc. Int. Conf. Image Processing 421 (2001), used partial low- 
frequency extraction to reduce the computational complexity of the transcoder of 
Assuncao et al. 

Natarajan et al, A Fast Approximate Algorithm for Scaling Down Digital Images 
in the DCT Domain, IEEE Proc. Int. Conf. Image Processing 241 (1995), proposed a 
fast algorithm for spatial resolution reduction in the DCT domain. This algorithm can 
be used to modify the transcoder of Assuncao et al as shown in Figure 4a to obta.n a 
frequency domain transcoder with spatial-resolution reduction. In Figure 4a the top 
row of operations is MPEG-2 processing and the bottom row of operations .s MPEG- 
4 processing. The MC stage implements frequency-domain motion compensate, 
and the Downsample stage performs spatial-resolution reduction in the frequency 
domain. However, this approach to frequency-domain transcoding wastes 
computational cycles because the MPEG-2 decoder performs a computat.onally 
expensive MC operation at the high SDTV resolution. 

Instead, based on the observation of Mokry et al, Minimal Error Drift in 
Frequency Scalability for Motion-Compensated DCT Coding, 4 IEEE Tran. Cir. Sys. 
Video Tech 302 (1994), that the MC and Downsample stages are interchangeable, 
Vetro et al. Minimum Drift Architectures for 3-Layer Scalable DTV Decoding, 44 IEEE 
Cons Elec. 527 (1998), suggested the transcoding scheme shown in Figure 4b, 
again with the top row of operations for MPEG-2 processing and the bottom row of 
operations MPEG-4 processing. In this frequency-domain transcoder, the frequency 
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domain frames are downsampled to the low C.F resolution an then mow 
compensated in the OCT domain. Because the computational expense MC ste e 
is performed at the tower C,F resotution, the compufafiona, complex,* ,s ^ 
reduced Two separate MC stages are required because the decoder and encoder 
teve different frame formats: the MPEG-4 encoder supports only I- and P-frames, but 
the MPEG-2 decoder also uses B-frames. Subsequently. Vetro et a,., Generated 
Motion Compensation for Drift Minimization, SPIE Conf. Vis. Comm. Image 
Processing (vol.3309 1998). Yin et al. Drift Compensation Architectures and 
Techniques for Reduced Resolution Transcoding, SPIE Conf. Vis. Comm. Image 
Lessing (vol.4671 , 2002), and Shen et al, A Very Fas, Video Spatial Resoluho 
Reduction Transfer, IEEE Proc. In.. Conf. ASSP 1989 (2002), proposed vanants of 
th e frequency-domain Iranscoder depicted in Figure 4b. However, these memo s 
are computationally oomplex because .he downsampled ./P-frames are upsampled 
before motion compensation to reduce drift. 

Subsequently, Liu e. al, ci.ed in .he background, demonstrated reduced- 
complexity frequency-domain .ranscoding also of the Figure 4b type. Although , the 
transcoder of Liu e. al is 50% more memory efficient and 70% less computabonally 
complex than other approaches, it has two significant disadvantages: (1) the 
frequency domain motion-compensation method uses an 800 Kb lookup table h i s 
, mp radical for DSP implementation, and (2) only progressive prediction formats are 
decoded efficiently; field prediction is computationally expense. 

The first preferred embodiment frequency-domain transcoding methods also 
use a Figure 4b type transcoder with input an MPEG-2 bitstream and VLD, IQ, and 
frequency domain downsampling followed by frequency domain inverse modon 
compensauon (reconstruction) to convert all inter blocks to infra blocks The tnfra 
frames are Ihen encoded by a frequency domain MPEG-4 encoder .ha. outputs the 
franscoded MPEG-4 bitstream. And to overcome drawbacks (1,-(2) of the transcoder 
of Liu et al, the preferred embodiment methods (1) use a macroblook shared 
information method similar to .he Song e, al method cited in me background and (2) 
have separate frame/field prediction approaches as illustrated in Figure 1b. 
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In particular, for the first preferred embodiment frequency domain 
downsampling methods frame-DCT block downsampling differs from field-DCT block 
downsampling. For frame-DCT blocks, downsample the blocks in the frequency 
domain similar to Merhav et al, cited in the background. This method performs 
vertical downsampling by a frequency-domain operation that is equivalent to spatial 
averaging of the top and bottom fields of each block. Horizontal downsampling is 
achieved by a frequency-domain operator that averages the spatial-domain even- 
and odd-polyphase components of each row. 

For field-DCT blocks, the top and bottom field DCT blocks are provided 
separately in MPEG-2. So first downsample horizontally separately for the DCT 
blocks of the top- and bottom-fields again with a method similar to that of Merhav et 
al cited in the background. Next, downsample vertically by averaging the 
horizontally-downsampled top- and bottom-field DCT blocks. Applying different 
downsampling operators to the frame-DCT and field-DCT blocks yields a frequency 
domain downsampling method that efficiently computes the DCT of the field- 
averaged, horizontal polyphase-component averaged input. Since top and bottom 
fields of interlaced video are highly correlated, the field-averaged DCT blocks may be 
used for frame-prediction as well as for field-prediction. Experiments show that very 
few noticeable artifacts arise after performing motion compensation on the field- 
averaged DCT blocks. These artifacts occur in the field-predicted blocks that have 
top- and bottom-fields that differ significantly. To prevent the propagation of any such 
artifacts in the encoder, the preferred embodiment methods may store the location of 
field-predicted blocks. During the encoder's mode-decision stage, blocks with mobon 
vectors pointing to field-predicted blocks are coded as intra blocks. This prevents 
any artifacts in field-predicted blocks from propagating to subsequent frames. Th,s 
method of preventing artifact propagation is a simplified implementation of Vetro et 

al.'s intra-refresh technique. 

For a more explicit version of the foregoing, again presume the four inter- 
ceded macroblocks „. ». «. » form a 2x2 quartet of macroblocks and were MPE6- 
2 compressed to yield the four motion vectors v,, n vs. v 4 together with the 
corresponding quantized 8x8 DCTs; the number of DCTs depends upon the 
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macroblock format six for 4:2:0, eight for 4:2:2. or twelve for 4:4:4. For eaoh n the 
motion vector v„ was determined by searching to minimize the prediction error, *„ of 
the 16x16 luminance part, of macroblock*. That is, the motion vector v n locates 
the predicted 16x16 luminance block A, from the prior reconstructed reference frame 
which minimizes the 16x16 predion error * = » - Now, each 16x16 * can be 
viewed as a quartet of 8x8 prediction errors: *,„ *,„ M end the correspond 
quantized 8x8 DCT blocks, £„,, «* are four of the 8x8 DCTs that were 
generated by the MPEG-2 compression. Let * denote the 16x16 block composed of 
the four 8x8 fig, arranged in the same pattern as the «* «V form «, 

Of course, if macroblocks *,, * 2 , *3, » were intra-coded, then there would be 
no motion vectors and the luminance parts, „. n. » would each be viewed as a 
quartet of 8x8 luminance blocks (y„ as the quartet a" d each ** ls 
transformed (8x8 DCT) to Y,, for encoding. Similar DCT blocks come from the 

chrominance blocks. 

The approach of Liu et al for downsampling in the frequency domain by a 
factor of 2 in each dimension converts the quartet of (reconstructed) macroblocks, *. 
* 2 * 3 , *,, into a single macroblock * which is to be MPEG-4 compressed as follows 
M for each o, the four 8x8 DCTs, fig, (k = 1 ,2,3,4), from *. take only the upper left 
(low'frequency) 4x4 DCT coefficients, and combine these four 4x4s to form a Single 
8x8 DCT block, E, Then these four DCT blocks (n = 1,2,3,4) are taken as E, the 
DCT blocks for the prediction error e of the luminance part , of downsampled 
macroblock*. For intra-coded frames the same approach applies, but using the 
luminance in place of the luminance prediction error; namely, for eaoh of the four 8x8 
DCT blocks, r„ (k - 1,2,3,4), from r. take only the upper left (low frequency, 4x4 
DCT coefficients, and combine these four 4x4s to form a single 8x8 DCT block Y 
Then these four 8x8 DCT blocks (n = 1 ,2,3,4) are taken as Y, the DCT blocks for the 
16x16 luminance part, of downsampled macroblock*. Again, the chrominance 

blocks are treated analogously. 

As illustrated in Figure 1b, the first preferred embodiment frequency domain 
methods downsample in the frequency domain by adapting the downsampling to the 
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incoming prediction forma. (frame-DCT blocks or field-DCT blocks for MPEG-2) as 

follows. . ... 

Frame-DCT blocks. Presume four 8x8 blocks x 2 , * 3( x* in the spatial 

domain which are .ocated as a 2x2 array forming a 16x16 block that is to be 

downsamp,ed by a factor of 2 in each dimension to yield an output 8x8 block * 

blocks may be either prediction errors (residuals) of an inter-coded picture or b ocks 

of pixels of an intra-coded picture. The preferred embodiment downsamp tag first 

averages pairs of pixe.s in the vertical direction and then averages pairs of the prior 

averages in the horizonta. direction. This can be written in 8x8 matrix format as: 

X = ( G1 *1 fil* + 01 X* <ti + & *3 fil' + Q2 X4 Q2 X )/4 

where superscript t denotes transpose and the 8x8 matrics and fe are: 



01 = 



1 1 0 0 0 0 0 0 

ooiioooo 
ooooiioo 

ooooooii 
oooooooo 
oooooooo 
oooooooo 
oooooooo 



and Qz - 



1 1 0 0 0 0 0 0 

ooiioooo 

0 0 0 0 1 1 0 0 
0 0 0 0 0 0 1 1 

oooooooo 
oooooooo 
oooooooo 
oooooooo 



Note that the left multiplication by ft averages pairs vertically and that the nght 
multiplication by &< averages pairs horizontally. Now let X k denote the 8x8 OCT of* 
that is, X R = S*ir< where S is the 8x8 OCT matrix. Because s is orthogonal, r - 
and S 1 is explicitly given by: 
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Further, let and U 2 denote the frequency domain versions of & and Q 2 , 
respectively; that is, ^ = SQ^S^' and U 2 = SQ 2 ST\ 

Now taking the DCT of the foregoing spatial domain downsampling expression 
yields the corresponding frequency domain downsampling expression: 

X = ( ffi X: 1 1 + t/i X 2 U 2 X + U 2 XzUS + U 2 Xa U 2 x )/4 
Thus the four input 8x8 DCT blocks (X k ) determine the downsampled output 8x8 DCT 
block (X) by matrix operations with the U, matrices. This approach has low 
computational complexity due to the possibility of factoring the matrices to simplify 
the matrix operations. In particular, make the following definitions: 
X +++ =Xi+X 2 + X 3 + X 4 
X + - = Xi +X2-X3-A4 
X_ + _ = Xi -X2 + X3-X4 
X_ + = Xi-X 2 -X 3 +X 4 
Note that these combinations require at most only eight additions/subtractions per 
frequency component. Then, with these combinations the expression for X becomes: 

X= ( U.X^ U: + U.X+- US * 17- + U.X-+ U} )/16 

where U+ = U, + U 2 and U. = U, - U 2 . These two combination matrices factor as U* = 
DPS, B 2 F, BSBSrW and U. - DPBi B 2 F- B 2 % ^d ' where the matrices 
D P B\ B 2 , F_, and F + are listed in the following; this factoring provides for fast 
computations and ultimately derives from Arai et al, cited in the background. Note 
that D is a diagonal 8x8 matrix and the off-diagonal 0s have been omitted for clarity. 
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Field-DCT blocks. The 16x16 luminance part of a macroblock in field-DCT 
coding consists of two horizontally-adjacent 8x8 blocks which make up the top field 
(16 columns by 8 rows) and the two corresponding 8x8 blocks of the bottom fie. d o 
the resulting four 8x8 OCT blocks consist of two from the top field and ^o from the 
bo ttom field. Reconstruction vertically interlaces these blocks after IDCT. More 
particularly, denote the four 8x8 luminance field blocks as a? 'a. * , * 4 wh.ch, 
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when interlaced, form a 16x16 block that is to be downsampled by a factor of 2 .n 
each dimension to yield an output 8x8 block x. Again, these blocks may be e.ther 
inter-coded field prediction errors or intra-coded field pixels; and denote the 
corresponding 8x8 DCT blocks as *** which are encoded in the 

MPEG-2 bitstream. The preferred embodiment downsampling first averages pa.rs of 
pixels in the horizontal direction and then averages the top and bottom fields. That ,s: 

Q2)i2 

xbo , = ( x »o, z e ; + x w & t )/2 

x S (j«w +x A< ")/2 
Again, to have this downsampling in the frequency domain, apply DCT: 

A* < "=(A* < "3t/i t + A*'"4^2 t )/2 

X = (A"* + A*")/2 
And as previously noted, the matrices factor to simplify the computations. In 
particular, U, = DPB, B 2 MA, A 2 A, Q, A>W V irW* W where 
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After the downsampling in the frequency domain, the Figure 4b transcoder 
structure requires (inverse) motion compensation (reconstmcton) in the frequency 
domain which converts intended frames/ftelds into intra-coded framesffields ,n 
order to then apply MPEG-4 encoding (with estimated motion vectors as in se«*on 2). 
The preferred embodiments use an inverse moton compensafton method wh.ch 
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takes advantage of correlations between blocks ot a macroblock to lower 
computational cost; see the Song et al reference in the background In parhcular, 
Figure 3d illustrates prediction of each of 8x8 blocks Q M > Q T . Q U <™ •»» 
corresponding 16x16 anchor blocks M. N. T. and U where M is made up of the four 
8x8 blocks Mo, M„ Mt and M 3 ; N is made up of the four 8x8 blocks No, N„ N 2 , and 
No- and analogously for T and U. As Figure 3c shows, the 1 6x16 anchor blocks have 
common 8x8 blocks: M, is the same as No, and Mo, *. T,. and Uo are all the same 
8X8 block; et cetera. Now the single motion vector for Q locates the 8x8 reference ,n 
M for Q M the 8x8 reference in N for Q N , the 8x8 reference in T for Q T > and the 8x8 
reference in U for Q u . Thus the horizontal and vertical displacements of the 8x8 
reference for Q M within M are the same as the displacements of the Q N reference 
within N, the displacements of the Q T reference within T, and the displacements of 
the Q u reference within U. This identity of displacements allows for rearrangement of 
the inverse motion compensation computations as follows. 

First some notation: let P re , denote an 8x8 reference block made from the four 
neighboring 8x8 blocks Po, P,. Pa, P* 'his can be written in 8x8 matrix format as P„ 
= £o«a SjiPjSio with S„ and S ]2 8x8 matrices like: 

['(•-.«. °<l-M>-> 0 , R n = 
L n - T a 

l nxn v nx(8-*) _ 

where I„ is an m identity matrix and 0 to is a fa. 0 matrix. For example, for S„ of 
the form U and S, 2 of the form R„, S„P,S J2 is an 8x8 matrix with the lower right n* m 
block the same as the upper left nm block of P, and the remaining elements all equal 

"° With this notation, Q M = Io*3 S„M,S j2 for appropriate Sjk (determined by the 
motion vector) and Q N = l<*a S„N,S, with the same * because of the same reiahve 
locations in the reference macroblock (same motion vector). Similarly, Q andQ 
also use the same S„ This reflects the four 8x8 blocks making up the macroblock Q 

ail have the same motion vector. 

Next these four sums can each be rewritten by adding and subtracting terms; 
and this can reveal duplicative computations among the four sums. In particular, 
Q M = losjsa SjiMjS j2 



0(8_„)jc(8-«) 0«(8-n) 
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= Soi(M 0 - Mi - M 2 + M 3 )S 02 + S 0 i(Mi - M 3 )P° 
+ P 1 (M2-M 3 )So2 + P 1 M3P°- 
where P° = S 02 + S 12 is a permutation matrix because S 02 and S 12 move columns in 
opposite directions and have complementary size, and similarly P 1 = S 01 + S 31 s 
another permutation matrix. Similarly, Q N yields 
Q N = lo^s SjiNjS i2 

= Soi(Ni - No - N 3 + N 2 )Si 2 + S 0 i(No - N 2 )P° 
+ P 1 (N 3 -N 2 )Si 2 + P 1 N 2 P 0 .. 
And due to No = M, and N 2 = M 3 , the second and fourth terms of this sum are the 
same as second and fourth terms in the sum for Q M . which will allow reuse of 
computations in the following. 
Analogously, 

Q T = Iosj* 3 SjiTjS j2 

= S 2 i(T 2 - T 3 - To + T 1 )S 02 + S 2 i(T 3 - Ti)P° 
+ P 1 (To-T 1 )S 02 + P 1 TiP 0 - 

and Q U = lo^s SjiUjS j2 

= S 2 i(U 3 - U 2 - Ui + U 0 )Si 2 + S 2 i(U 2 - U 0 )P° 
+ P 1 (Ui-Uo)Si 2 + P 1 U 0 P 0 .. 
Now to compute DCT(Q M ), DCT(Q N ), DCT(Q T ), and DCT(Q U ), which are the 
four prediction error DCTs, begin with DCT(Q M ) and use the similarity transform 
nature of the DCTto have 
DCT(Q M ) = 

DCT(Soi){DCT(M 0 ) - DCT(Mi) - DCT(M 2 ) + DCT(M 3 )} DCT(S 02 ) 
+ DCT(Soi){DCT(Mi) - DCT(M 3 )}DCT(P° ) 
+ DCT(P 1 ){DCT(M 2 ) - DCT(M 3 )}DCT(S 02 ) 
+ DCT(P 1 )DCT(M 3 )DCT(P 0 ).. 
Second, compute DCT(Q N ) , 
DCT(Q N ) = 

DCT(SoiKDCT(Ni) - DCT(N 0 ) - DCT(N 3 ) + DCT(N 2 )} DCT(S 12 ) 
+ DCT(Soi){DCT(N 0 ) - DCT(N 2 )}DCT(P°) 
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+ DCT(P 1 HDCT(N 3 ) - DCT(N 2 )}DCT(Si 2 ) 
+ DCT(P 1 )DCT(N 2 )DCT(P°).. 
And as previously noted. No = M, and N 2 = M* so in the second line of the 
expression for DCT(Q») the DCT(So,KDCT(No) - DCT(N 2 »DCT(P°) has already been 
computed as DCT(S 01 ){DCT(M,) - DCT( M 3»DCT(P° ) in the second line of DCT(Q ). 
Similarly, the fourth line of DCT(Q N ). DCT(P<)DCT(N 2 )DCT(P°). is the 
fourth line of DCT(Q M ), DCT(P')DCT( M 3)DCT(P°>. Thus the computation of DCT(Q ) 
can reuse computations from DCT(Q M ). 

Third compute DCT(Q T ) noting that To = M 2 and T, = Ms, so the computations 
can use the equalities P'(T. - T,)S 02 - P'W, - ^ and P'T,P° = P'M 2 P°, and 
thereby reuse computations from DCT(Q ). 

Fourth, compute DCT(Q U ). Initially, note that U 0 = Ti and U 2 = T 3 , so use 
S 21 (U 2 - Uo)P° = S 21 (T 3 - Ti)P° and P 1 U 0 P° = P^P* and thus reuse terms from the 
third computation. Lastly, note that U 0 = N 2 and U, = N 3> so P 1 (U 1 - U 0 )S 12 = P (N, - 
N 2 )S 12 and thus reuse the term from the second computation. 

5 Motion Vector Refinement in the Frequency Domain 

Section 2 described how MPEG-4 motion vectors may be estimated for the 
downsampled macroblocks from the MPEG-2 motion vectors contained in the input 
bitstream. After the estimation, a half-pixel motion-vector refinement has been shown 
to improve the reliability of the estimate. However, such a refinement is difficult to 
implement in frequency-domain transcoders that use the scheme outlined in Figure 
4b Plompen et al., A New Motion-Compensated Transform Coding Scheme, IEEE 
Proc int Conf. ASSP (1985),. and The Performance of a Hybrid Videoconferencing 
Coder Using Displacement Estimation in the Transform Domain, IEEE Proc. Int. 
Conf ASSP (1986), suggested a method for frequency-domain motion estimation 
that may also be used for frequency-domain motion-vector refinement. However, 
because their method is based on the Hadamard transform, it is not as 
computationally efficient for frequency domain motion-vector refinement. More 
recently, Liang et al., in cross-referenced patent application No. 09/089,290, filed 
6/1/1998 and published 12/26/2002, proposed a fast algorithm for frequency-domain 
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m o,ion-vec,or refinement. However, this method is oomputationally expensive when 
the macroblock is aligned with reference DCT blocks. 

The preferred embodiment motion vector refinement methods apply < * 
Figure 4b freguency-domain transcoder that caputs an MPEG-4 bfistreanv the 
MPEG-4 encoder input is a seguence o, frames comprised of 8x8 ,nt* 
The first frame is encoded as an .-frame and each successive frame becomes a P- 
irm!Lispredic,edfrom,eprecedingfiame. 
derive a motion-vector estimate and corresponding figure of ment for each 
m acroblock. ,f the figure o, merit indicates a poor motion-vector esfimate then 
perform a 0.5 pixel mofion-veCor refinement as explained below. To 
Ition-vecor estimate for a pedicular 16x16 macroblock, *e cogent OCT ^blocks 
(f0 ur for luminance and one or more for each chrominance are IDCT and *e 
lon-vector esfimate is used to co-locate the macroblock agarnst the DC ^bloc 
, he preceding reference frame, as depicted in Figure 5a. If the reference DCT blocks 
ITby the macroblock are IDCTd, then an 18x18 search window may be used 
,or me bilinear interpolation that precedes a 0.5 pixel motion-vector refinement. 
Unfortunately, .his straightforward approach is computationally «P™~« 
consequently, the preferred embodiment methods provide a reduced-complexrty 

implementation as follows. rv-rhinrksin 

The alignment of the gray macroblock against the reference DCT blocks 
Figure 5a creates three cases of interest. In Figure 5a, o (P> measures the 
di placemen, of the upper (left, macroblock boundary from the nearest, cov „ A 
upper (left, boundary of a reference 8x8 DCT block. The firs, case 
situation in which file macroblock is no, aligned with any reference DCT block 
Idaries; therefore, 8 > a > 0, 8 > 6 > 0 and nine reference DCT blocks are 
oledbythemacroblock. Now define an 18x18 search window whose al^me, 
aaains, the reference DCT blocks is described by a and b, where a = a 1 and - P 
1T search window also covers only nine reference DCT blocks and file prxels 

as described a, the end of the section. Using this search window and the 
madtl, perfomi a half-pixe, mofion-veCor refinement The refined mofion veCor 
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indicates the portion of the search window that is subtracted from the macroblock to 
obtain residual blocks which yield the P-frame macroblock after a DCT operation. 

In the second case, a = 0 and (3 > 0 so that the upper boundary of the 
macroblock is aligned with a reference DCT block boundary, as shown in Figure 5b. 
Here the macroblock covers six reference DCT blocks. Set a = a + 1 and b = B + 1 to 
define an 18x18 search window as in the first case, then twelve reference DCT 
blocks will be covered by the search window. Even with Liang et al.'s fast algorithm, 
computing the IDCT of all these reference blocks to obtain the pixels in the window is 
expensive. To reduce the complexity, the preferred embodiment refinement methods 
set a = 0 and b = B + 1 thereby obtaining a 16x18 search window whose upper 
boundary aligns with the reference DCT boundary. Now use Liang et al.'s fast 
algorithm to recover the search-window pixels from the six reference DCT blocks 
covered by the window. Next, symmetrically extend the top and bottom of the search 
window to obtain an 18x18 window. Implement the symmetric extension by creating 
new top and bottom rows that are copies of the old top and bottom rows respectively. 
This symmetric extension technique is justified if the image is smooth along the 
search window boundaries. Finally, refine the motion vector using the 18x18 search 
window as explained in the first case. 

in the third case, a =0 and B =0 so that the upper and left boundaries of the 
macroblock are aligned with reference DCT block boundaries, as shown in Figure 5c. 
Four reference DCT blocks are covered by the macroblock. On setting a = a + 1 and 
b = B + 1 to define an 18x18 search window as in the first case, 16 reference DCT 
blocks would be covered by the search window. Computing the IDCT of these many 
blocks is prohibitive. Once again, to reduce the complexity, set a = b = 0 to obtain a 
16x16 search window that covers four reference DCT blocks. The search window 
pixels are obtained by applying IDCTs to the four DCT blocks. As in the second 
case first symmetrically extend the top and bottom of the search window to obtain a 
18x16 search window. Next, symmetrically extend the left and right boundaries of the 
search window by copying the old left-most and right-most columns to obtain the new 
left-most and right-most columns of a 18x18 search window. This search window » 
now used for motion refinement as in the first case. 
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The Liang et al method for obtaining pixel values in corner subblocks of 8x8 
blocks from the 8x8 DCT blocks uses the DCTs of cropping matrices which define 
these corner subblocks and proceeds as follows. 

The operation on each 8x8 block involved in a reference macroblock .s either 
(1 ) obtain all of the pixels in the block or (2) crop the block so that only the P ,xels 
needed remain. In matrix terminology, the operation of cropping a part of a block can 
be written as matrix multiplications. For instance, cropping the last . rows of an 8x8 
matrix A can be written as - CA where ft is the 8x8 matrix with all elements 
equal to 0 except CO J) - 1 for 8-™ *J s 7. Similarly, with ft the 8x8 matrix with all 0 
elements except C M > - 1 for 8- <-j <- 7, post-multiplicatior, by ft crops the last . 
columns. Thus the operation of cropping the lower right * rows by » columns 
submatrix of A can be written as /I™, = CiAC* 

Now denoting the 2-D DCT of A by A means A = S 1 AS where S is the 8x8 
DCT transformation matrix. Thus A„ - fti" A SC.. And tben denoting the product 
as V and ft* as T implies A„ -UAt. Note that the first 8-m rows of V are 
all zeros and the first 8-„ rows of T are all zeros. Thus denoting the «rt matnx of 
the . nonzero rows of V as U c and the 8x„ matrix of the . nonzero rows of T as 2c, _ 
the mxn matrix W consisting of the cropped portion of A is given by A^ = Uc A 
Tc'. Actually, Uc is the last m rows of the inverse 8x8 DCT matrix, and Tc is the last n 
rows of the inverse 8x8 DCT matrix St. 

And a 16x16 reference block for the motion vector searching is assembled 
from the pixels of these cropped subblocks. The first case of Figure 5a would have 
one full 8x8 IDCT plus eight cropped blocks. And the IDCTs have fast computation 
methods by using a factorization of the DCT matrix as follows. First, note that the 
8x8 DCT matrix 5 = HUIUM where these 8x8 factor matrices are the same 

as those of section 4. 

After applying the foregoing fast DCT on the columns and then applying the 
cropping matrix, only m nonzero rows exist. The computation for the row DCT then 
takes only 42m operations. Also, either A croppei or W could be computed, so the 
total computation amounts to 336 + 42min(m,«) operations. 
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Alternative preferred embodiment methods refine the motion vector for a single 
target NxN block which has an NxN reference block lying within a 2x2 array of 
reference frame NxN blocks; this corresponds to considering just one of the four 
blocks of the macroblocks in the foregoing. Again, if the reference block does not 
align with the blocks of the reference frame, then have a search window by 
expanding the reference block one row/column on each side. But if the reference 
block does align with a block of the reference frame, then again pad on the al.gned 
sides to create the search window. 

6 Fast, Drift-Free Transcoding 

The foregoing sections 4 and 5 describe preferred embodiment methods that 
improve the performance of frequency-domain transcoders which are based on the 
framework depicted in Figure 4b. Although these methods make effective use of 
computational and memory resources, frequency-domain motion compensate » 
difficult to implement. Moreover, because frequency-domain motion compensation 
must be invoked twice in the transcoder, the gain from the elimination of the 
IDCT/DCT blocks is small. In addition, frequency domain downsampling techniques 
result in frames that differ significantly from the original resolution frames. When 
these altered frames are used for motion compensation, drift artifacts result. Sect.on 
4 proposes a reduced-complexity implementation of Vetro et al.'s intra-refresh 
technique to mitigate drift artifacts. Now this section shall provide computationally 
efficient preferred embodiment transcoding methods that eliminate drift arfifacts. 
Section 8 shall demonstrate that the new transcoding methods may be used to 
implement a multi-format transcoder. 

To eliminate the drift artifacts in frequency-domain transcoders based on the 
framework of Figure 4b, first observe that frequency-domain downsampling 
algorithms use frequency-domain operators to perform horizontal and vertical 
averaging followed by decimation. For interlaced video sequences, vertically 
averaged fields may differ significantly from the top and bottom fields. This causes 
severe drift artifacts because motion compensation must be performed specifically 
from the individual fields. Therefore, to eliminate drift, vertical averaging should be 
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avoided. To downsample interlaced frames in the spatial domain without vertical 
averaging, Ghanbari advocates extraction of the top field of each frame followed by 
averaging of even and odd-polyphase components along every row (averaging with 

respect to the column index). 

The preferred embodiment drift-free methods effectively extract the top field in 
the frequency domain followed by horizontal averaging in the spatial domain. The 
Downsample-IDCT stage of the preferred embodiment transcoder illustrated in Figure 
6 performs the method. The Downsample-IDCT stage is an IDCT implementation 
that functions differently for B-frames and for anchor l-/P-frames as follows. 

For B-frames, first downsample frame-DCT blocks vertically with a de- 
interlacing one-dimensional (1-D) IDCT that outputs the top field of each frame-DCT 
block in the spatial-frequency domain (frequency domain for the horizontal 
dimension, spatial domain for the vertical dimension). Section 7 explains an 
implementation of the de-interlacing 1-D IDCT. Next, apply a 1-D IDCT to each of the 
rows of this top field and then horizontally downsampled by either (a) averaging the 
even- and odd-polyphase components of each row in the field or (b) dropping the 
odd-polyphase component of each row. The latter approach to horizontal 
downsampling is faster but may produce slightly perceptible artifacts. 

(For B-frames with field-DCT blocks, the first downsampling is just selection of 
the top field DCT followed by a vertical IDCT and then one of the horizontal 

downsampling methods.) 

For l/P-frames (frame-DCT blocks), apply 2-D IDCT to the DCT-blocks to 
convert to spatial domain, and then horizontally downsample using one of the 
approaches as previously described for the B-frames: either horizontal averaging or 
odd phase discarding. Vertical downsampling for l/P-frames is postponed because 
both top and bottom fields of the l/P-frames are required during the subsequent 

motion compensation. 

(For l/P-frames with field-DCT blocks, apply 2-D IDCT and then a horizontal 
downsampling for both top and bottom field blocks; again postpone vertical 
downsampling until after motion compensation.) 
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After the B-frame vertical and horizontal downsampling and the l/P-frame 
horizontal downsampling, perform inverse motion compensation (reconstruction) to 
convert inter blocks to intra blocks as follows. 

For B-frames, only the top fields are motion compensated using either the top 
or bottom field of the horizontally downsampled l/P-frames. 

For P-frames, perform usual motion compensation. Then vertically 
downsample the l/P-frames by discarding the bottom fields of these frames. 

The thus-decoded (reconstructed), spatially-downsampled frames are fed to 
an MPEG-4 encoder which generates the output bitstream using motion estimation 
with re-used motion vectors as illustrated in Figure 6. Following section 7 describes 
the de-interlacing 1-0 IDCT that enables efficient B-frame downsampling. Of course, 
bottom fields instead of top fields could be selected. 



7. De-Interlacing 1-D IDCT 

As described in section 6, the frequency-domain transcoding scheme depicted 
in Figure 6 provides fast, drift-free transcoding because expensive frequency-domain 
motion compensation is avoided and vertically averaged fields are not used for 
motion compensation. To implement this scheme, the Downsample-IDCT stage must 
directly extract the spatial-domain even polyphase components (top field) from B- 
frame frame-DCT blocks. This extraction is efficient because unwanted polyphase 
components are not computed. The following explains how to implement an IDCT 
method mat extracts polyphase components from frame-DCT blocks. Suppose that x 
is a length-TV data sequence and z is the tf-point DCT of x. Denote the even- and 
odd-polyphase components (each length NI2) of x by x, and x„, respectively. Let z, 
and r, represent the even- and odd-polyphase components of z in bit-reversed order, 
respectively. In particular, for yV = 8: 
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Now the expression of me N-point OCT in terms of the M-point OCT (see Hou 
re ,erenoe in the background) reia.es * to * through T(N), an * x Af decima«on-,n-Ume 
DCT matrix, as follows: 

T(N/2) T(tf/2) 
KT(W/2)Q -KT(N/2)QJ 





= N/2 







T(2) = 
Nil x Nil 



(zo is scaled by V2 for notational convenience); Q is a 



where the matrix on the rigl 

1 1 
cos(;r/4) -cos(;i74) 
diagonal matrix: diag[cos((4m + 1)n/2iV)] form = 0, 1, ...,M2-1;and 

K = RLR 1 , where R is the bit-reversal permutation matrix; and L is the Nil x Nil 

lower-triangular matrix: 

"1 o 0 0 ••• 0" 
-12 0 0 — 0 
L= 1 -2 2 0 - 0 

• • * 

-12-2 2 

Matrix inversion (the DCT matrix is orthogonal, so inversion is transposition) shows 
that the polyphase components of x are given by 

T'(AT/2) QT'(JV72)K' " 
T'(iV/2) -QT'(iV72)K\ 



V 


= 2/N 







extracted from the DCT block by 

x e = T t (M2)z P + QT t (M2)K t z. 
For N - 8, u -tW * ♦ QlW *. and the 4-poin, iDCT, T'(4), requires 4 adds and 
9 multiplies using the Lee decomposition. Multiplication with K requires 6 adds and 5 
shifts while replication with Q requires 4 multiplies. Note that the two 4-pomf 
IDCTs in the equation for x, may be performed in parallel. 

More explicitly for N- 8, the de-in.erlacing 14) IDCT may be found as follows. 
First, the 14) 8-point IDCT, using the abbreviation cN = cos(N*/16), is: 
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Then consider only the even indices of x, and appy the 2n periodicity of the cosine, 
c(N+32) = cN, to have: 
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Note that the <2 has been moved from the matrix into the z 0 component. Next, 
separate the even and odd indices of z to yield: 
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Using the symmetries of the cosine. cN = c(32-N) and cN = -c(16-N), plus reverse- 
bit ordering the z components gives: 
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"i 
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c2 
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x 4 
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c2 
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-c2j 



,/V2 



L 



+ y 2 
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cl 
c5 
-c7 
L-c3 



c5 c3 c7 

c7 -cl c3 

-c3 c5 cl 

cl c7 c5 



The first 4x4 matrix is just the 4-point 1-D iDCT matrix; and as previously noted, the 
second 4x4 matrix factors into the product of three factors: (1 ) a diagonal matnx of 
cosines, (2) the 4-point 1-D IDCT matrix, and (3) a simple matnx K: 



cl c5 c3 c7~ 

c5 c7 -cl c3 

-c7 -c3 c5 cl 

-c3 cl c7 c5 



1 c 4 c2 c6 

1 -c4 -c6 c2 

1 c 4 -c2 -c6 

I - c 4 c6 -c2 



"cl 0 0 0 
0 c5 0 0 
0 0 -c7 0 
0 0 0 -c3 

Now K = RLR where R is the (symmetric) 4-point bit-reversal permutation matrix, and 
L is the 4x4 lower diagcnal matrix of ±1 and ±2 e.ements which arise from the 
coefficients in the iterative application of the angle addition formula for the cos.ne, 
c(2N+1) = 2c(2N)d-c(2N-1): 



R = 



Thus 



10 0 0 
0 0 10 
0 10 0 
[0 0 0 1. 

K = 



and L = 



1 
1 

-1 
-1 -2 





" 1 0 


0 


0 




-1 2 


0 


0 




1 -2 


2 


0 




-1 2 


-2 


2 


0 


0 0" 






2 


-2 0 






0 
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2 2j 



This factoring provides a fast computation mathod for the second 4x4 matrix in terms 

of the 4-point 1-D IDCT matrix. 

The foregoing 8-point de-interlacing IDCT applies in the fast, dnft-free 
preferred embodiment transcoder of section 6 as follows. 
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First vertically downsample the B-frame frame-DCT blocks by top-field 
extraction from each 8x8 OCT biocR using the de-interlacing 1-D IDCT on each of the 
columns; this yields 8-oolumn x 4-row blocks having frequency-domain row index and 

spatial-domain column index. 

Next, perform horizontal downsampling by one of the following two prefened 

embodiment methods: 

(1 ) averaging the even- and odd-polyphase components of each of the four 
top-field rows by firs, applying an 8-poin. 1-D IDCT ,o each of the four top-field rows 
,o convert to spatial-domain column index and then averaging the even- and odd- 
polyphase components to yield the downsampled 4x4 in the spatial domain, or 

(2) eliminating the odd-polyphase component of each of the four top-field rows 
by applying the de-intehacing 1-D IDCT to each of the four top-field rows to yield the 
downsampled 4x4 in the spatial domain. As mentioned in section 6, the second 
method is faster but may produce slightly percepfible artifacts around sharp verfical 

edges. . . . 

More explicit let Z denote an 8x8 frame-DCT of 8x8 spatial bloc* X wh,ch 
may be either a block of pixels (intra-co<led) or a block of prediction errors (inter- 
coded). Then the overall downsampling is: 

(a) For B-frames: first apply the de-interlacing 1-D IDCT with respect to the row 
index to each of the columns of Z to extract M«, the 8-column x 4-row top-field of X 
but still with column index still in the frequency domain: 

m\ - T'(4) i r + QT'(4)K' l", for k = 0, 1 7 

where k is the column Index. 8x8 Z is the interlace of 8x4 Z, and 8x4 Z, after reverse 
bit-ordering, Z„ - [z°, i,\%.-}f. *M,andM, = [m°.,.., .J. 

(b) Next, for method (1) first apply 8-poin. 1-D IDCT to each of the rows o. M M. 
»o yield 8X4 top field X,, and then average pairs of pixels in the rows to yield the 4x4 

downsampling of X. m 

For method (2) for each of the four rows of 8x4 Me, apply the de-interlacng 1-D 
,DCT with respect to the column index to directly yield the 4x4 downsamplmg of X: 
jk 4<( = . T(4) n k , + QT*(4)K' n k , for k = 0, 1 , 2, 3 
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where n\ and *, are .he bit-reverse ordered even- and odd-polyphases 0 , n' which 
ls ,he transpose o. the K» row of M, and A, is the transpose o, the Kth row of W 

8. Multi-Format Transcoder „ w „i„,hp 
in applicattons such as video streaming, content is usually avertable ,n the 
MPEG-2 in daced format. However, each end-user may demand 
streams should be deiivered to him/her in one of several available standards such as 
MP G 4 H 263, Windows Media Player, or Real Video. To support this repu.remen 
Tmulforma, transcoder tha, can convert an MPEG, bitstream - 
standard is critical. This sec«on explains how to efficiently rmplemen, a mu Mm* 
transcoder based on the foregoing Fast, Drift-Free (FDF, transcoder ,n section 6. 

each standard that the end-user may demand. Thus, ftrs. mod,fy me MPEG-2 
decoder so that it provides de-interiaced, spatially-downsampled raw frames w,.h 
abated motion-vector informal as described in section 6 and shown ,n F.gure 
6 The required modifications are listed below. 

1 . Replace the 2-D IDCT stage of the MPEG-2 decoder with the Downsample- 
IDCT stage used in the fast drift-free transcoder of sections 6-7. 

2 Modify the MPEG-2 decode MC stage so that it motion compensates 
h0 ri Z onta,ly-downsam P led .VP-frames. For B-frames, perform motion compens^n 
on the horizontally-downsampled top fte,d only. After B-frame motion compensate 
discard the bottom fields of the associated anchor l-/P-frames. 

3 Use one of the methods in Section 2 to estimate motion-vectors for the 
downsampied frames. After moving the MPEG-2 decoder as described above, .he 

that re-uses me es.ima.ed motton vectors provided by the modifte MPEG-2 ^deooder. 
T„ operate me multt-forma, .ranscoder. feed me inpu. content to the modrfted MPEG- 
2 deLer tha. now outputs de-interlaced, spatially-downsampled, raw frame a ong 
with estimated motton-vectors. Then inpu. the frames and motion vectors to he 
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specified standard. Incorporating the transcoding algorithms in the decoder 
implementation thus provides fast, drift-free multi-format transcod.ng. 
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